Bug Fix for apache-airflow-providers-jenkins JenkinsJobTriggerOperator#22802
Merged
potiuk merged 6 commits intoApr 12, 2022
Merged
Conversation
…it should not fail the task, instead it should continue polling until reaches the max allowed polling tries
apache-airflow-providers-jenkins
apache-airflow-providers-jenkinsapache-airflow-providers-jenkins JenkinsJobTriggerOperator
uranusjr
reviewed
Apr 7, 2022
try except accounts for only the risky part of the code
uranusjr
reviewed
Apr 7, 2022
uranusjr
reviewed
Apr 7, 2022
uranusjr
approved these changes
Apr 7, 2022
Contributor
|
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
potiuk
approved these changes
Apr 12, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is to eliminate a bug in jenkins job trigger opertor. During polling for the newly created build, it is possible to get a failure and it is causing the task to fail and if task have retries then it will result in duplicate builds which could lead to problems.
here is the exception I'm receiving in the task logs:
[2022-04-05, 18:50:32 PDT] {jenkins_job_trigger.py:176} INFO - Triggering the job Segmentation/Production/Downstream Trigger - Production on the jenkins : JENKINS with the parameters : None [2022-04-05, 18:50:32 PDT] {base.py:70} INFO - Using connection to: id: JENKINS. Host: server.com, Port: 443, Schema: , Login: ****, Password: ***, extra: True [2022-04-05, 18:50:32 PDT] {jenkins.py:43} INFO - Trying to connect to [https://server.com:443](https://server.com/) [2022-04-05, 18:50:32 PDT] {kerberos_.py:325} ERROR - handle_other(): Mutual authentication unavailable on 403 response [2022-04-05, 18:50:33 PDT] {jenkins_job_trigger.py:154} INFO - Polling jenkins queue at the url https://server.com/queue/item/5870086//api/json [2022-04-05, 18:50:33 PDT] {taskinstance.py:1700} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/jenkins/__init__.py", line 575, in jenkins_request return self._response_handler( File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/jenkins/__init__.py", line 536, in _response_handler response.raise_for_status() File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://server.com/queue/item/5870086//api/json During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task self._execute_task_with_callbacks(context) File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks result = self._execute_task(context, self.task) File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task result = execute_callable(context=context) File "/opt/bitnami/airflow/dags/git_dags/jenkins_custom_lib/operators/jenkins_job_trigger.py", line 185, in execute build_number = self.poll_job_in_queue(jenkins_response['headers']['Location'], jenkins_server) File "/opt/bitnami/airflow/dags/git_dags/jenkins_custom_lib/operators/jenkins_job_trigger.py", line 156, in poll_job_in_queue location_answer = jenkins_request_with_headers( File "/opt/bitnami/airflow/dags/git_dags/jenkins_custom_lib/operators/jenkins_job_trigger.py", line 51, in jenkins_request_with_headers response = jenkins_server.jenkins_request(req) File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/jenkins/__init__.py", line 589, in jenkins_request raise NotFoundException('Requested item could not be found') jenkins.NotFoundException: Requested item could not be found [2022-04-05, 18:50:33 PDT] {taskinstance.py:1267} INFO - Marking task as UP_FOR_RETRY. dag_id=segmentation_v1, task_id=trigger_downstream_jenkins, execution_date=20220406T002500, start_date=20220406T015032, end_date=20220406T015033 [2022-04-05, 18:50:33 PDT] {standard_task_runner.py:89} ERROR - Failed to execute job 326683 for task trigger_downstream_jenkins Traceback (most recent call last): File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/jenkins/__init__.py", line 575, in jenkins_request return self._response_handler( File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/jenkins/__init__.py", line 536, in _response_handler response.raise_for_status() File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://server.com/queue/item/5870086//api/jsonat the line 589 of
jenkins_requestin Jenkins library it returns aJobNotFoundExceptionexception which leads to failure ofjenkins_request_with_headersmethod. Because poll_job_in_queue doesn't account for exception handling it will fail and ultimately sets the whole task for retry (if configured!). This will lead to creating another build by going over the whole processes which is problematic.Instead it should complete its polling retries and if failed then raise airflow exception and ultimately set the task for retry.